Optimal Code Length Based Cost for Unsupervised Grammar Induction

An effective grammar can be induced from natural language sentences by simultaneously minimizing the cost of encoding the grammar and the cost of encoding the corresponding derivations of the sentences. In previous work, the cost of encoding a derivation was computed in terms of the number of bits it requires to encode which of the possible productions is used to expand each of its non-terminals. However, this ignored the fact that if some productions are used more often than others in the derivations, then they could be encoded with fewer bits using optimal code length based encoding. This paper presents a new derivation cost that uses such an encoding and applies it for inducing grammars. Minimizing this new derivation cost also corresponds to maximizing the probability of the derivations. Thus besides being theoretically more appealing, experimental results on sentences from clinical reports show that this new derivation cost also leads to induction of grammars that have better parsing performance.